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SUMMARY 

Purpose 

The primary objective of this study was to conduct a systematic evaluation of the effectiveness of 
adjunct instructional materials, when used in a relatively unstructured environment. Previous studies of 
adjunct materials have been conducted in relatively structured environments. In most of these studies, 
subjects were given short passages with adjuncts and were not allowed to refer back to the text. 

Method 

Four experiments were conducted which were, in effect, two experiments replicated with minor 
modifications at two institutions. Over 400 Air Force ROTC students served as subjects. 

In experiments one and two, junior AFROTC students at two universities were given adjunct mate- 
rials as part of thf reading assignments outside the classroom for a one semester course in aerospace studies. 
The student text contained 74 articles in 700 pages. Selected articles were assigned for student reading with 
each chapter. The materials for each chapter were distributed so that approximately one third of the 
subjects received adjunct materials containing multiple choice questions, one third of the subjects received 
adjunct materials with constructed response questions, and one third of the subjects received no adjunct 
materials. The adjunct materials were questions and answers keyed to the text. Students were instructed to 
read the text before attempting to answer the questions. There were an average of 1.75 adjunct questions 
per page of text. Assignment of the adjunct materials was rotated in a counterbalanced order so that each 
subject completed approximately one third of the chapters under each experimental condition (multiple 
choice questions, constructed response questions, or control). Performance was measured by administering 
a criterion test after completion of each reading assignment. The tests contained from 43 to 67 questions 
measuring performance directly covered by adjunct questions (relevant questions) and from 7 to 23 ques- 
tions measuring performance not directly covered by the adjunct questions (incidental questions). The 
incidental questions in most cases required the students to apply the facts and principles contained in the 
text. 

In experiments three and four, senior AFROTC students at the two universities completed a ten 
chapter, 56 page pamphlet as part of a two-week section on military justice. The same instructions for 
adjunct use were given to these students. The average adjunct question count was 1.75 per page of text. 
Half of the classes received adjunct materials for the first five chapters. The remaining classes received 
adjunct materials for the last five chapters. The multiple choice adjunct materials were used in one study 
and constructed response questions were used in the other. A50-item criterion test was administered after 
the students had completed all ten chapters. Separate scores were obtained for the incidental items (20) and 
relevant items (30) under each experimental condition. 

Results 

Analysis of the data indicated that: 

1. For three of the experiments, scores on the relevant test items of the criterion tests for students 
who received either type of adjunct questions were significantly higher than the scores of the 
control group. In one experiment the results were not significant. For this experiment, there 
were indications from the responses to the questionnaire that some of the subjects may have 
violated the rule of not using the adjunct materials while serving as a cortrol. 

2. In all four experiments, subjects did not perform significantly better on measures of incidental 
learning when they used adjunct materials than when they did not use adjunct materials. 

3. The responses of the, subjects to the questionnaire indicated generally favorable attitudes 
toward the adjunct materials. 

The results on the incidental tests may, in part, be explained by differences between the adjunct 
question and the incidental test items. The adjunct items were designed for retention of facts and prin- 
ciples, but the incidental test questions required application of facts and principles. If the adjunct questions 
had been application items, results may have been different. 
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Tlic findnigs indicated that tlic following recommendations should be made: 

1 . Adjunct mstructional materials should be utilized in classroom settings for relevant itcmn. 

2. Further research is required concerning various questionLused as adjuncts^nd criterion items. 
The role of application questions is particularly iniportanFfor further investigition. 
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THE EFFECTS OF ADJUNCT INSTRUCTIONAL MATERIALS EMPLOYED 
OU'reiDE THE CLASSROOM ON THE PERFORMANCE OF AIR FORCE ROTC STUDENTS 



I. INTRODUCTION 

For many years, researchers in the area of instaictional technology have sought to develop techniques 
which will increase the effectiveness of learning from textual materials. A number of techniques have been 
developed which have been shown to increase the learning efficiency of the student. The most widely 
accepted and used of these techniques is programmed learning. Programmed learning has been shown to be 
an effective technique for promoting learning from text. One of the disadvantages of programmed learning 
IS that the development of programmed instructional materials is expensive. A more economical technique 
thouglu to have many of the advantages of programmed learning is the use of adjunct instructional 
materials. Althougli this technique is not new, it has received increased attention in recent years. 

In this context, adjunct instructional materials arc questions, and usually answers, that parallel the 
text. In normal use of adjunct instructional materials, the student is instructed to read the text, answer the 
questions, and then check his answer (immediate feedback). Questions may be provided within the text 
(e.g., at the end of each page or subsection) or at the end of each major section or chapter. The questions 
also may be printed and packaged separately from the text such as in a workbook. The latter feature makes 
possible the relatively inexpensive application of the adjunct process to existing textual materials. Although 
the use of adjunct materials represents a promising technique, additional research is needed to identify the 
most effective procedures and to identify the types of learning situations fpr which adjunct materials are 
likely to be effective. 

Background 

The technique of using adjunct type questions dates back to early research perfonned by Pressey 
(1926, 1927). Pressey developed a simple mechanical device which administered and scored multiple choice 
questions. It also provided the student with immediate knowledge of results. He proposed the use of the 
device as the basis for a self-instructional system which would incorporate the basic laws of learning. 

The potential economy of using adjunct questions with standard textual materials has led to recent 
interest in the use of adjunct instructional materials. This interest has produced a series of studies into the 
nature and use of adjunct materials (Rothkopf, 1966, Rothkopf and Bisbicos, 1967; Erase, 1967; Erase, 
1968). These studies have shown under laboratory conditions that the use of adjunct materials does 
facilitate the learning of textual material. 

The Rothkopf and Erase studies have been the subject of sharp criticism from Carver (1972). Carver 
has criticized these studies for being based on a weak theoretical friunework and a methodological approach 
which provides poor evidence to support generalization to practical classroom usage. Carver has critici/.ed 
the studies on the following points: 

1. Failure to maximize the similarity between the experimental and practical situations. A review 
of the studies cited above reveals that the subjects were either paid volunteers, or students, 
participating as a class requirement. The experiments were conducted in a laboratory or similar 
environment. 

2. Failure to permit the student to refer back to previously covered materials or questions. In a 
practical learning environment, the student nomially would be given this freedom, 

3. Failure to control for running time. Carver notes that higher performing groups required a 
longer time to complete the materials. He suggests that the better performance may be a 
function of the increased time and not the treatment. 

Another criticism tliat can be made of the Rothkopf and Erase studies is that the duration of the 
expenments has been rather short. Typically the subjects are asked to read a relatively sliort text (2,000 to 
9,000 words) and respond to a post-test the same day. Short experiments of this nature have been criticized 
by Shulman (1970). The question may be asked. What effect would having the experiment extend over a 
longer period of time (such as a semester) have on the findings? 



7 



ERLC 



10 



Air Force Research 



To some extent, research conducted by the Air Force on adjunct instructional materials has offset the 
criticisms of Carver. Meyer (1965) compared the effectiveness of a training program utilizing adjunct 
materials with conventional classroom instruction for providing refresher training in aircraft weapons. A 
group of aircrew members were ^ven a self-study work book containing adjunct questions, the correct 
answers, and references to the manual containing the answer. Subjects were permitted to refer back to the 
manual as necessary. Students were instructed to complete the self-study program on their own time 
(average time 253 minutes). A second group of aircrew members was given conventional classroom instruc- 
tion (time: 250 minutes) on the subject matter. The group receiving the adjunct materials scored signifi- 
cantly higher on a criterion test administered following completion of training. 

Yasutake (1974) used Air Force ROTC students in a study of the effectiveness of adjunct materials 
for aiding learning of textual materials. Four experimental groups and one control group were used. 
Students in the experimental groups were provided booklets containing adjunct questions for use with their 
text books. Booklets for the four experimental groups varied in terms of the type of question (multiple 
choice or constructed response) and presentation of questions (for use at the end of each page or at the end 
of each chapter). Referral back to the text was permitted, and time for study was controlled (200 minutes, 
in the classroom). Yasutake found thai students using adjunct materials performed significantly better on 
relevant criterion 'est questions (questions directly covered by an adjunct question). However, he found no 
significant differences in the performance of the control and adjunct groups on incidental criterion test 
questions (materials not directly covered by the adjuncts). Multiple choice and constructed response 
adjunct questions were found to be equally effective. Placement of adjunct questions at the end of the 
chapter was found to be as effective as placement at the end of each page. 

Research Needs 

The Yasutake study has demonstrated that the use of adjunct instructional materials can improve 
student performance on relevant test items when students are given the freedom to refer back to the text as 
necessary. However, proposals for use of the technique in practical situations would be strengthened by 
further studies in which the artificial control of running time is eliminated. Although this control was 
necessary to offset the Carver criticism, it is artificial in the sense that students normally control their own 
reading time while studying. A study is needed to test the effects of adjunct materials in a practical 
situation which is as near **real world" as possible. A study of this type would allow variables which 
normally vary from one instructional situation to another to vary naturally. Preferably, such a study sfiould 
be replicated in several situations using materials for more than one subject matter in a systematic replica- 
tion (Sidman, I960). The research described in this report was designed to partially meet this need. 



The purpose of this study was to conduct a systematic evaluation of the effectiveness of adjunct 
materials when used in a relatively unstructured environment. The primary goal was to determine the effect 
on student performance of employing adjunct materials outside the classroom. Previous studies of the 
effectiveness of adjunct materials have been conducted in higlily controlled classroom environments* This 
study was designed to essentially replicate the Yasatake study (1974) without the strict classroom controls 
used in that study. The study was conducted in an operational environment to demonstrate the reliability 
and generality of cariicr findings with adjunct instructional materials in a practical situation. Regular classes 
were employed and students were allowed the option of referral back to the prose instructional materials. 

- The basic design of the study was systematically replicated in four settings. The replications were 
made using student populations at two universities (Texas A and M University and Brigham Young Uni- 
versity), two divergent .kinds of subject matter (aerospace history and military justice), and two course 
lengths (two weeks and one semester). Findings similar to those of earlier studies sliould provide evidence 
of the usefulness of adjunct materials in practical situarions. 



II. PROBLEM 



Puq)Ose 
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Objectives 

The specific objectives of this investigation were to determine the answers to tlie following questions 
under conditions approximating the usual instructional environment: 

1. Do students receiving adjunct materials perform better on measures of relevant and incidental 
learning than students not receiving such instruction? 

2. Are adjunct materials using constructed response questions more effective in terms of siudent 
performance than multiple choice questions? 

3. Do the students using adjunct instructional materials follow the general procedures suggested? 

4. What is the general attitude of students toward the use of adjunct instructional materials 
outside the classroom? 



III. METtiOD 

Subjects 

The subjects for the study were junior and senior college students enrolled in the Air Force Reserve 
Officer Training Corps (AFROTC) programs at Texas A and M University (Texas A&M) and Brigham 
Young University (BYU) during the fall of 1972. At Texas A&M, 143 juniors and 124 seniors participated 
in the study. At BYU, 66 juniors and 109 seniors participated. The total subject population was 442. 

Materials 

Aerospace Studies 300 Instructional Materials, A two-volume text, Readings in Growth andDeveiop- 
ment of Aerospace Power, served as a basic source book for students in Aerospace 300 at both universities. 
The 700 page text contains 74 articles which range in length from 2 to 20 pages. Students were assigned 
selected articles from the text as homework and tested periodically on the assigned materials. 

An analysis of the content of the text was made to identify important items of infomiation. Then, 
adjunct instnictional materials (questions with text references and a separati> sheet with the answers) were 
developed with the intent of requiring students to demonstrate their understanding of the important items 
of infomiation. Each adjunct question was prepared in two formats, multiple choice (MC) and constructed 
response (CR). An average of 1.75 adjunct questions were prepared for each page of instructional materials. 
Two sets of adjunct questions (either MC or CR), the answers, and page references were then assembled 
into packets with the questions on the first pagc(s) and the answers and references on the following page(s). 
The adjunct materials were prepared under contract by McDonnell Douglas Aircraft Corporation. 

Aerospace Studies 300 Evaluation Materials, The contractor who prepared the adjunct questions also 
developed the evaluation instrument. It consisted of a pool of 287 test items covering material in the 
74 articles of the text. Of these, 202 concerned information specifically covered by one of the adjunct 
questions. These items were classified as relevant test items. Approximately half of these questions were 
identical to the adjunct questions. The other half were roughly equivalent to the adjunct questions but with 
minor modifications. The remaining 85 questions concerned materials contained in the reading but not 
specifically covered by adjunct materials. These items usually required an application of the facts or 
principles in the text to a hypothetical situation. They were classified as incidental test items. All test items 
were presented in the multiple (:hoice format. 

Aerospace Studies 400 htstrttctional Materials. The book, the Military Justice System, was used as a 
text for Aerospace Studies 400 at both universities. The book contains 56 pages of textual materials 
supported by 100 pages of appendices concerning the uniform code of military justice. The procedures used 
to develop adjunct materials for Aerospace Studies 300 also were used to prepare adjunct questions for the^ 
military justice text. A total of 98 adjunct questions were prepared (1.75 per page) in the MC and CR 
formats. The adjunct materials were assembled into two packets of questions, answers, and references. Each 
packet contained adjunct materials for approximately half of the test (five chapters). The adjunct materials 
were developed by the same contractor who developed the materials for the AS 300 course. 

Aerospace Studies 400 Evaluation Materials. Twenty -rour incidental test items were developed to 
measure knowledge of materials not directly covered by adjunct questions. As in Aerospace 300, the 
incidental test items differed from the relevant test items, in that rnost incidental test items required an 
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application of facts or principles, while the relevant test items required the direct recognition of a specific 
fact or principle. A 50 item evaluation instrument was then prepared by selecting 30 items from the 
questions used as adjunct materials and 20 items from the pool of incidental test items. Half of the relevant 
items and half of the incidental items were from the first half of the text. The remaining items covered the 
second half of the text: All test items were presented in the multiple choice format. 

Adjunct Materials Instructions. A set of instructions was developed and issued to all students before 
the adjunct materials were assigned. The instructions recommended tliat the student read the assignment, 
answer the adjunct questions, correct the answers, and then review missed questions. The detailed instruc- 
tions are in Appendix A. 

Adjunct Materials Questionnaire, A questionnaire was developed to assess the extent to which the 
students followed the instructions to read the text before answering questions and to avoid using adjuncts 
when serving as a control group. The questionnaire also included questions designed to measure student 
attitude toward the adjunct materials. The Adjunct Materials Questionnaire is in Appendix B. 

Experimental Design 

A basic experimental design was developed and replicated systematically by varying the location, 
subject matter, and course length. The basic experimental design provided for comparison of the per- 
formance of students on criterion tests when they used adjunct materials with the performance of the same 
.students when they did not use adjunct materials. This was accomplished by assigning-units of reading 
materials for which adjunct materials were provided and units of reading materials for which adjunct 
materials were not provided. The materials were assigned as homework. Assignments were made in a 
counterbalanced order so that each student used adjunct materials part of the time and served as a control 
subject at other times. After reading assignments had been completed, the students were given criterion 
tests to measure their retention. The criterion tests contained both relevant and incidental test items. The 
relevant and incidental test items were scored separately to provide measures of relevant and incidental 
learning. 

In effect, four experiments were conducted. Two experiments were conducted at each institution, 
one using juniors and one using seniors. For convenience in describing the experiments and results, each 
experiment was assigned an identification number as indicated in Table 1 . 



Table i. Criterion Tests 



Experiment 


Class 


Institution 


Subject f 


1 


junior 


Texas A & M 


Aerospace History 


2 


junior 


BYU 


Aerospace History 


3 


senior 


Texas A &M 


Military Justice 


4 


senior 


BYU 


Military Justice 



Procedures. 

Experiment I and 2. The juniors at both institutions employed adjuncts for one semester. The 
experiipenter conducted a conference with ROTC faculty members at both universities and secured agree- 
ment to utilize the adjunct instructional materials with all members of the junior class. The class at Texas 
A&M was divided into ten sections with four instructors. The class at BYU was divided into four sections 
with two instructors. Students at Texas A&M were assigned 24 of ^the 74 articles. Students at BYU were 
assigned 40 articles. 

An assignment matrix was developed for each ROTC department. Reading materials were divided into 
units. Sections were assigned to treatments by units in a counterbalanced order. Assignments to treatments 
were rotated so that one third of the units were completed under treatment one, MC (multiple choice); one 
third under treatment two, CR (constructed response); and one third under treatment three, C (control, 
reading materials without adjunct questions). A quiz was administered following each unit. Texas A&M was 
scheduled to administer six quizzes and BYU was scheduled to administer nine quizzes. However, adminis- 
trative problems were encountered at Texas A&M which prevented strict adherence to the planned pro- 
cedures. On one occasion, students inadvertently were not given the adjunct instructional materials 
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according to the plan. This group was used as an. additional control group for data analysis. The staff at 
Texas A&M also decided to eliminate one of the quizzes. This required an additional adjustment in the data 
analysis procedures. 

Students were assigned chapters in the text at the beginning of each unit of instruction. At the time 
the assignments were made, the students who wwe to use adjunct materials were given packets containing 
either the MC or CR adjunct materials and instructions for their use (Appendix A). Students in the control 
groups were encouraged to avoid using the adjunct materials ^ven to students in the other sections. 

At the conclusion of each reading assignment, a ten item quiz was administered. The ten items for 
each quiz were selected by the using university from the pool of items. An attempt was made to include 
some incidental questions on each quiz. Five quizzes of 10 items each were given at Texas A&M with a total 
of 43 relevant items and 7 incidental items. Nine quizzes of 10 items each were given at BYU with a total of 
67 relevant items and 23 incidental items. For Texas A&M, each quiz score served as a separate measure of 
the effectiveness of the treatment. It was necessary to treat each .score as a separate measure since an 
unequal number of quizzes was administered under each treatment (due to the elimination of the one quiz). 
For BYU, the quiz scores for each treatment were totaled to provide the measure of the effectiveness of 
each treatment. 

After each quiz, the students were rotated to a new treatment and given the appropriate materials. 
After the final quiz, the students were given the anonymous questionnaire (Appendix B): 

Experiment 3 and 4, Seniors at Texas A&M and BYU were given adjunct materials for use with the 
military justice block of instruction (approximately two weeks). The class at Texas A&M was divided into 
six sections with two instructors. The class at BYU was divided into four sections with two instructors. The 
sections were divided and assigned to treatments so that half of the sections received adjunct materials for 
the first half of the text and half of the sections received adjunct materials for the second half of the text. 
Each section served as a control group for the portion of the course for which adjunct materials were not 
provided. MuUiple choice adjunct materials were provided at BYU and constructed response adjuncts were 
provided at Texas A&M. Instructions for the use of adjunct mate rials (Appendix A) were given at the time 
the materials were distributed. Students who did not receive adjuncts were encouraged not to use the 
adjunct materials provided to the other sections. 

At the end of the military justice block of instruction, a 50 item test was given at both universities. 
Thirty items in each test were relevant and 20 items were incidental, instructors at each institution selected 
the test items for local use to conform with the 30/20 mix of relevant/incidental items. The test items were 
divided equally between the first half and second half of the military justice reading assignment. There were 
15 relevant items and 10 incidental items covering the first half of the reading assignment, and a like 
number of items covering the second half. The anonymous questionnaire was administered immediately 
following administration of the criterion test. 

Data Analysis 

Transformation of Scores. Since different tests were i.sed to measure achievement under the different 
experimental conditions, it was necessary to transform the test scores into a standard measure before 
comparisons could be made. This was accomplished by tr ansf orming the test, scores into slandard scores. 
The standard scores were further transformed into T scores with a mean of 50 and a standard deviation of 
10 to simplify computation and to avoid the use of negative numbers. 

Regression Analysis. In an experim'ent of this type, variables other than the experimental variable can 
influence the criterion measures. This may result in experimental error which can lead to erroneous 
interpretations of the results of the experiment. There are statistical techniques which may be used to 
control the effects of these types of variables. Therefore, it is important that statistical controls are used for 
as many of these variables as possible. In the present study, two types of variables were identified as having 
possible effects on performance. These were subject variables (aptitude, motivation, study habits, etc.) and 
situational variables (instructor, time of day, etc.). Multiple regression technkjues provide effective methods 
for statistically controlling the effects of these types of variables. Multiple regression was used to analyze 
the data collected in this study. 

Using multiple regression techniques, the problem of determining the effect of a variable (such as a 
treatment) is approached by detennining what portion of the variance of the criterion scores is accounted 
for by the variable. This is accomplished by developing two multiple regression equations (or models) to 
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predict the criterion scores. The first model (called the full model) includes infomiation on the variable of 
interest and any control variables. The second model (called the restricted model) is the same as the full 
model except that information on the variable of interest is omitted. The squared multiple correlation 
coefficient (RSQ) obtained by using each model to predict the criterion represents the percent of the 
variance accounted for by the variables included in the model. Thus, the percent of variance accounted for 
by the variable of interest can be obtained by subtracting the RSQ for the restricted model from the RSQ 
for the full model (i.e., RSQf- RSQr = percent of variance accounted for). A statistical test can then be 
used to determine if the percent of variance accounted for is statistically significant. When the percent 
variance accounted for is statistically significant and the variable is categorical (such as treatment or group 
membership) the conclusion may be made that the treatment means are significantly different.* 

Separate regression analyses were made for each experiment. Details of the analyses and the specific 
regression models are presented in Appendix C. The general questions of the study were restated in more 
specific terms to provide a basis for developing the analyses. The analyses were designed to answer the 
following questions: 

Question 1: Is there an interaction between treatments and sections for relevant items of the 
criterion test (i.e., is the effect of each treatment the same for all sections)? 

Question!: Is there an interaction between treatments and sections for incidental items of the 
criterion test (i.e., is the effect of each treatment the same for all sections)? 

Question 3: Are the mean scores of the subjects on the relevant items of the criterion tests signifi- 
cantly different when the subjects have used multiple choice or constructed response 
adjunct materials than when they have not used adjunct materials? 

Question 4: Are the mean scores of subjects on incidental items of the criterion tests significantly 
different when the subjects have used multiple choice or constructed response adjunct 
materials than when they have not used adjunct materials? 

Questions: Are the mean scores of subjects on the relevant items of criterion tests significantly 
different when the subjects have used multiple choice adjunct materials than when they 
have used .const aicted response adjunct materials? 

Question 6 . Are the mean scores of subjects on the incidental items of the criterion tests significantly 
different when the subjects have used multiple choice adjunct materials than when they 
have used construldted response adjunct materials? 

Additional Analyses. A further analysis was made to provide additional information for use in 
evaluating the efi'ccts of the adjunct materials. In this analysis, the proportions of the variance of the 
criterion scores accounted for by two additional variables, section and instruction, were computed. 
Information on these variables was indirectly included in the regression analysis as part of the control 
variable (subject). However, it was not possible to determine the influence of the section and instructor 
variables from this analysis. Thus, the additional computations were required. The proportions of the 
vanance were computed from additional regression models. The models and procedures used are described 
in Appendix C. 

The Adjunct Materials Questionnaire (Appendix B) was analyzed by recording the propor^tions of 
responses relating first to conformity with directions and second to attitudes toward adjunct materials. The 
analyses were designed to answer the questions posed by objectives 3 and 4, Section 2. 



IV. RESULTS 

Effects of Adjunct Materials 

The effects of the adjunct materials on performance were evaluated by comparing the mean scores for 
subjects when they used adjunct materials with their mean scores when they did not use adjunct materials. 
The results obtained f rom the relevant and incidental criterion test items are discussed separately below. 

Mean scores were computed for each treatment condition for each section. Examination of the means 
revealed that for Experiment 1 (Texas A&M, Aerospace Studies) the means for one section (Section Three) 

* For a more thorough discussion of this approach see Wnrd and Jennings (1973) or Kellcy et al (1969). 
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deviated markedly from the other section means. Such a large deviation may indicate that uncontrolled 
factors influenced performance in that section which did not influence the pcrfomiance in the other 
sections. Since the presence of uncontrolled factors in a single section would contaminate interpretation of 
the results, a decision was made to eliminate the data from section three from the analysis. 

Relevant Test Items. The mean scores of the subjects on the relevant test items of the criterion tests 
are presented in Table 2. Examination of the table reveals that the mean scores for the subjects were 
consistently higher when adjunct materials were used. Comparison of the results for the multiple choice and 
constructed response adjunct materials used in Experiments 1 and 2 indicated that the subjects* scores were 
higher for both types of adjunct materials. 



Table 2 Mean Scores by Treatment for Relevant Test Items 



Treatment 






Experiment 




1 


2 


3 


4 


MC 


52.94 


52.49 




53.53 


CR 


49.57 


51.80 


50.87 




C 


47.30 


45.72 


49.12 


47.47 



The regression analysis procedures described in Section 3 were used to test the significance of the 
differences between the observed means. The results of this analysis arc presented in Tables 3 and 4. Tests 
were made to determine if an interaction exists between section and treatment (Question 1, Section 3). The 
results of these tests are presented Jn Table 3. As may be seen from the table, the F values obtained were 
not significant at the .05 level of confidence. 



Table 3. Results of the Tests of the Significance of the Interaction 
Between Treatments and Sections for Relevant Test Items 



Experiment 


^^9f 




Pet. var. 
Acct. for 


-F 


df 


p 


1 


.3764 


.3490 


2.74 


.85 


24/465 


.6701 


2 


.6049 


.5616 


4.33 


1.47 


9/121 


.1648 


3 


.7461 


.7139 


3.22 


.65 


20/103 


.8622 


4 


.7204 


.7180 


0.24 


.08 


1/ 98 


.9997 



Table 3 presents the results of the tests of the significance of (lie differences observed between the 
treatment means in the four experimental situations (Question 3, Section 3). Significant differences (at the 
.05 level) were observed for Experiments 1, 2, and 4. For Experiment 3, the differences between means is 
not significant at the .05 lev6T of confidence. However, the level of probability associated with the F value 
obtained (P = .0568) is only slightly above the .05 level. 



Table 4, Results of the Tests of Signifscance of the Differences 
Between Treatment Means for Relevant Test Items 



Experiment 


RSQf 


RSQ^ 


Pet. Var. 
Acct. for 




df 


p 


1 


.3490 


.3003 


4.87 


18.28 


2/489 


<.0000 


2 


.5616 


.4688 


9.28 


13.76 


2/130 


<.0000 


3 


.7139 


.7055 


.84 


3.61 


1/123 


.0568 


4 


.7180 


.5934 


12.46 


47.72 


1/108 


<.0000 



13 



Since only two treatments were used in Experiment 4, it is clear that the mean for the adjunct 
condition is significantly higher than the mean for the control condition. It is also clear for Experiments 1 
and 2 that the mean for the multiple choice adjunct materials is significantly higher than the mean for the 
control condition. However, it is not possible to tell from this analysis whether the means for the con- 
structed response materials are significantly higher than the means for the control condition, or whether the 
means -for the multiple choice adjunct materials are significantly higher than the means for the constructed 
response adjunct materials. An additional analysis was required to test the significance of the difference 
between these means. 

The significance of the differences between the paired means was tested using the Schaffe technique 
for multiple comparisons. The resuUs of this analysis are presented in Table 5. For Experiment 1, an F 
value of 12.00 was obtained for the comparison of the means for the multiple choice and constructed 
response adjunct materials. This F value is significant at the .01 level of confidence. An F value of 7.09 was 
obtained for the comparison of constructed response adjunct materials and control condition (Question 5, 
Section 3). This F value is significant at the .05 level of confidence. 

Table 5. Results of Scheffc Test for Multiple Comparisons 
for Relevant Test Items, Experiments 1 and 2 

Required Required Observed 



Experiment F' .05 F' .01 F 



1 A&M Aerospace 


(df = 2/488) 


■(df= 2/488) 




MC and CR 


6.04 


9.32 


12.00 


CR and Control 


6.04 


9.32 


7.09 


2 BYU Aerospace 


(df =^ 2/128) 


(df = 2/I28) 




MC and CR 


6.14 


9.56 


.24 


CRand Control 


6.14 


9.56 


18.16 



For Experiment 2, an F value of .24 was obtained for the comparison of the mean scores for the 
multiple choice and constructed response adjunct materials. This F value does not meet the requirement for 
significance at the .05 level. An F value of 18.16 was obtained for the comparison of the means obtained 
for the constructed response adjunct materials and the control condition. Tliis F value is significant at the 
.01 level of confidence. 

incidental Test Items. The mean scores of the subjects on the incidental items of tlie criterion tests 
arc given in Table 6. Examination of the table reveals that the means for the control condition are higher 
for three of the four experiments. In the first experiment, the mean for the constructed response adjunct 
condition was higher. 



Table 6. Mean Scores by Treatment for Incidental Test Items 



Treatment 




Experiment 






1 


2 


3 


4 


MC 


49.48 


49.40 




49.72 


CR 


51.18 


50.25 


48.93 




C 


49.48 


50.33 


51.06 


50.27 



The regression analysis techniques described in Section 3 were used to test the significance of differ- 
ences of the means for the incidental item test results. The results of the tests for interaction between 
treatment and section (Question 2, Section 3) are given in Table 7. As may be observed from the table, the 
tests for Experiments 1, 2, and 4 were not significant, indicating that the effects of the treatments were 
consistent across sections. The test for interaction for Experiment 3 is significant (p = .0182). Examination 
of the section means for each treatment revealed no consistent pattern. 
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Table 7, Results of the Tests of the Significance of the Interaction 
Between Treatments and Sections for Incidental Test Items 



Experiment 


RSQf 


RSQ^ 


Pet. Var. 
Acct for 




df 


p ' 


1 


.4002 


.3450 


5.52 


,90 


22/215 


.5962 


2 


.3888 


^ .3725 


1.63 


.36 


9/121 


.9520 


3 


.7021 


.5911 


11.10 


1.92 


20/103 


.0182 


4 


.6528 


.6367 


1.61 


.45 


1/ 98 


.9150 



The results of the tests of the differences (Question 4, Section 3) between the means for Experiments 
1, 2, and 4 are given in Table 8. The test for significance of differences between means was not made for 
Experiment 3 due to the presence of the interaction. Examination of the table reveals that the means 
obtained in Experiments 1,2, and 4 are not significantly different (at the .05 level of confidence.) 



Table 8. Results of the Tests of Significance of the Differences 
Between Treatment Means for Incidental Test Items 



Experimtnt 


RSQf 


RSQ^ 


Pet Var. 
Acct. for 


F 


df 


p 


1 


.3450 


.3380 


.71 


1.28 


2/237 


.2802 


2 


.3725 


.3707 


.18 


.19 


2/130 


.8316 


3* 














4 


.6367 


.6359 


.08 


.24 


1/108 


.6325 



♦Significance of the differences between means was not tested because of the observed interaction. 



Contributions of Treatment and Other Factors to Performance. To provide an additional basis for 
evaluatmg the relative effect of the treatments on perfomiance, the proportion of variance accounted for 
by three other variables- subject, instructor, and section were computed. This information is summarized in 
Table 9 for relevant test items and in Table 10 for incidental test items. Both tables indicate that the 
subject variables account for the largest proportion of the variance. For the relevant test items, the 
treatment variable, with the exception of Experiment 3, accounted for the next largest portion of the 
variance. 



Table 9. Proportion of Variance Accounted for by Variables 
Influencing Performance for Relevant Test Items 







Proportion of Variance Accounted for 




Source 


1 


2 


3 


4 


Treatment 


.0487 


.0927 


.0077 


.1247 


Treatment-Section Interaction 


.0274 


.0433 


.0329 


.0023 


Subjects* 


.3003 


.4688 


.7055 


.5934 


Sections 


(.0324) 


(.0750) 


(.0698) 


(.0662) 


Instructor 


(.0106) 


(.0409) 


(.0276) 


(.0521) 


Unexplained Subject Variance 


(.2679) 


(.3938) 


(.6357) 


(.5272) 


Unexplained Error Variance 


.6236 


.3951 


.2539 


.2796 



*Tutal variance due to subjects is composed of variance due to sections and unexplained subject variance. Variance due to 
instructor is included in variance due to sections. 
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Table 10. Proportion of Variance Accounted for by Variables 
Influencing Performance on Incidental Test Items 



Source 




Proportion of Variance Accounted for 






2 


3 


4 


Treatment 


.0044 


.0018 


.0114 


.0008 


Treatment— Section Interaction 


.0500 


.0162 


.1109 


.UloZ 


Subjects* 


.3522 


.3707 


.5747 


.6359 


Sections 


(.0376)^ 


(.0062) 


(.0733) 


(.0774) 


Instructor 


(.0245) 


(,0048) 


(.0381) 


(.0433) 


Unexplained Subject Variance 


(.3146) 


(.3645) 


(.5014) 


(.5585) 


Unexplained Error Variance 


.5933 


.6112 


.2979 


.3472 



*Total vanancc due to subjects is composed of variance due to sections and unexplained subject varianc*c. Variance due to 
instructor is included in variance due to sections. 



Questionnaire Results 

Conformity to Suggested Directions, The results of the questionnaire items (Appendix B) dealing 
with conformity to suggested directions are shown in Table 1 1. Students were free to utilize the adjunct 
instructional materials in a manner of their own choosing. Responses to the first statement indicate that 
student use of adjunct varied considerably among the experimental groups. Generally, the seniors (Experi- 
ments 3 and 4) tended to make greater use of the adjunct materials than the juniors. The juniors (Experi- 
ments 1 and 2) tended to utilize the adjunct materials for test study purposes (responses to the second 
questionnaire statement) more than the seniors. 

Conformity to the suggestion of reading the assignment before the adjunct questions varied between 
the experiments by a proportion of .53 and .66 (responses to the third statement). This suggested that 
slightly better than half of the students followed the suggested procedure. 

Subjects were encouraged to avoid the use of adjunct instructional materials when serving in the 
control group. There was a greater conformity to this direction at BYU than at Texas A&M (responses to 
the fourth questionnaire statement). Of particular interest is the relatively high degree of nonconformity by 
the seniors at Texas A&M (Experiment 3), and the high proportion of other responses by the seniors at 
BYU (Experiment 4). 

Attitude Toward Adjunct Instructional Materials, The results of student responses to the section of 
the Adjunct Materials Questionnaire (Appendix B) dealing with attitude toward the adjuncts are shown in 
Table 12. The first five statements are arranged in a descending order of positiiA^ affect toward adjunct 
materials (for the actual order of presentation, see Appendbc B). It may be seen that students were 
generally favorable toward adjunct materials with a proportion of between .74 and. .87 having a desire for 
such items in other classes (first response statement). Trfe remaining responses shift proportionally from yes 
to ne as attitude reflected-by-the statements changes com positive to negative with the lowest proportion 
of yes responses to the most negative statement (fifth/. The sixth response statement dealt only with the 
students receiving two types (MC/CR) of adjunct questions. The majority of students preferred the multiple 
choice (MC) adjuiaicl questions to the constructed response (CR). 

V. CONCLUSIONS, DISCUSSION, AND RECOMMENDATIONS 

Conclusions 

Analysis of the data collected in the four experiments suggests that the use of adjunct instructional 
materials in the typical ROTC classroom environment has the following effects: 

1. Students perform better on measures of relevant learning when they have used adjunct instruc- 
tional materials than when they have not used adjunct materials. 

2. Students do not perform better on measures of incidental learning when tliey have used adjunct 
instructional materials than when they have not used adjunct materials. 



16 



c 

o 
22 
S 

C/3 



c 

1 

a 

c 
o 
u 

c 
o 
a. 

0^ 



CO 



c 
o 

r 

o 
a. 
o 





other* 










c 




V 




E 








V 


Z 


a 




X 




u 






> 
















0 


« 








c 




•» 




E 








pt 


z 


X 




u 






M 








> 




« 




V 




JZ 








0 


N 








C 




« 




E 






o 


• 


Z 


a 




X 




u 










« 




>- 





« 












£ 








0 










c 




• 




E 






o 


m 


Z 


a 




X 




u 






M 




« 




>- 



o 



o 



o 



O 



O 
S 

CO 

to £ 
O C 

c 

> to 

-a CO 
«^ oo 

3 CO 



o 



CO 



o 
o 



00 



O 
O 



CO 



CO 

ro 



vo 
vo 



O 
O 



CO 



00 
00 



O 



CO 

vo 



vo 

CO 



O 
o 



o 
o 



o 
o 



OO 



1^ 


.2 


a> to 


<2 


to 




udy 




ave 
djur 






^ CO 






c> 


>^ 


*to 

CO 






CO 


C 

o 

to 
♦«» 


ding 
gat 


ion 
ked 


o 


CO C 




un 




'-^ 

CO 




>^ 








I use 
tests 


I rea 

befo 


Whei 
adjU] 



o 
c 

CO 

E 
o 



17 



20 



o 



o 



o 



o 
z 



00 

o 



00 



00 



00 



O 



o 



o 



o 



o 



o 
z 



o 

ON 



ON 



00 



00 



00 

o 



O 



O 



CI 

o 



o 



o 



o 



O 



00 



00 



ON 

00 



o 



o 

CO 



o 



o 



o 
o 



8 S 



8 8 



VO 



ON 
CO 



VO 
VO 



00 



00 
00 



00 



CO 



ON 

00 



u 

I" 

o o 
■*"• i-i 

2 o 
2^ *> 

> 3 



3 C 

§ 2 a 



c .5 

T3 3 

o .S 
•§ 

3 




•S ts i: 
2-2 o 

C <• o 

«> -a jc 

u- O 

C^jC o 



18 



21 



3. Neither the multiple choice nor constructed response fonn of adjunct question is consistently 
superior to the other for promoting learning. 

4. Student populations vary in the degree to which they follow recommended procedures for 
using adjuncts. 

5. Students have a generally favorable attitude toward the use of adjunct materials. 



Discussion 

Previous studies conducted under controlled conditions have shown that the use of adjunct instruc- 
tional materials can improve learning of relevant materials. The present study has verified the findings 
of the earlier studies under less controlled conditions. The results were verified in the more nearly "real 
world" environment of the university classroom and over a longer period of time (up to one semester). 
Replication of the results under the more practical conditions used in this study support generalization of 
the results to the classroom and help to justify use of the technique in the classroom. The fact that the 
basic experiment was replicated vnth four subject populations and two subject areas further support this 
generalization. 

The results of studies of the effectiveness of adjunct materials for promoting learning of incidental 
materials have been inconclusive. Some studies have reported increased learning of incidental materials 
when adjuncts are used. Other studies have found no significant differences. In the present study, adjunct 
materials did not significantly affect learning of incidental materials in any of the four experiments. A 
number of explanations may be offered to account for the failure of the adjunct questions to improve 
incidental learning. 

One explanation would be that the adjunct questions used in the study were not suitable for eliciting 
incidental learning. It may be that the use of a more comprehensive type of adjunct question is necessary to 
elicit incidental learning. To effectively produce incidental learning, it may be necessary for adjunct ques- 
tions to be constructed so that they require the student to think about the materials in general rather than 
simply recall specific facts or principles. Questions requiring students to think or apply their knowledge arc 
often called application questions. The incidental test items used in this study were application questions 
which required application of previously learned principles. However, the questions used as adjuncts were 
not . application questions and did not force the students to apply the knowledge acquired-while.they were 
still studying the materials. It could be argued that if application questions had been used as adjuncts, 
forcing the students to apply the knowledge while still fresh, incidental learning may have occurred. A 
study by Watts and Anderson (1971) provides support for this hypothesis. The results of that study 
indicated that application questions used as adjuncts are more effective than retention questions for 
promoting learning as measured by both retention and application criterion test questions. 

Another plausable explanation for the failure of the adjunct questions to elicit incidental learning is 
that they may not have been based on meaningful passages of the text or that the questions were not 
"meaningful." Rickards (1973) conducted an experiment in which "meaningful" questions (requiring 
organization of facts in relation to more inclusive concepts) were used as adjuncts. The meaningful ques- 
tions were found to be significantly more effective than adjunct questions which require rote learning of 
facts or ideas. 

Recommendations 

Although a significant amount of research has been conducted on the use of adjunct instructional 
materials, many questions remain to be answered. The following recommendations are made; 

1. Adjunct instructional materials should be employed in classroom settings to enhance relevant 
learning. 

2. A thorough review of the literature on learning from prose should be made. The review should 
include: (a) research of the effects of meaningfulness on learning and (b) research on the nature 
and effects of various types of questions employed as adjunct materials. 

3. Research should be conducted to clarify the effects of various types of questions. The research 
should include questions related to simple facts, questions designed to require the student to 
organize facts in relation to a more general concept (meaningfulness), questions designed to 
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require the student to apply factual knowledge, and questions designed to require the student 
to apply principles discussed in the text. 

4. Research studies should be systematically replicated using different subject populations and 
different types of subject matter. The research should be conducted in environments which will 
permit generalization to practical, everyday learning situations. 
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APPENDIX A: ADJUNCT MATERIALS INSTRUCTIONS 

INSTRUCTIONS FOR USE 

The adjunct questions you have received will help you learn your 
course material. Research has shown that you will gain the most if you 
follow the rules given below: 



1. Read your text assignment without looking at^ the adjunct questions, 

2. Immediately after reading the text, answer all the adjunct 
questions. 

3. Compare your answers with the answers in the adjunct materials. 

4. Review the text (at the page and paragraph indicated) for 
information concerning the questions you missed < 

5. Read the next reading assignment and then repeat the cycle. 
The chart below illustrates this process. 



READ THROUGH 
ASSIGNED. READING 



READ AND ANSWER 
ADJUNCT QUESTIONS 



COMPARE YOUR ANSWERS 
TO CORRECT ADJUNCT ANSWERS 



ADVANCE TO NEXT 
ASSIGNED READING 



I 



ARE YOUR ADJUNCT 
QUESTIONS CORRECT? 



YES 



I 



NO 



I 



IS REVIEW NECESSARY? 



NO 



REVIEW PERTINENT MATERIAL 4 



YES 



Following each adjunct question is a text reference in parenthesis. 
The first number in the parenthesis is the page and the second number is 
Che paragraph. The paragraphs are numbered from the page where they begin. 
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APPENDIX B; ADJUNCT MATERIALS QUESTIONNAIRE 

* 



DO NOT WRITE YOUR NAME ON THIS QUESTIONNAIRE. PLEASE READ ALL THE 
STATEMENTS BEFORE MAKING ANY ANSWERS, THEN ANSWER EACH STATEMENT BY 
PLACING AN "X" UNDER EITHER YES OR NO. IF YOUR ANSWER IS NEITHER 
YES NOR NO, select' THE ANSWER WHICH IS THE CLOSEST TO YOURS. 

STATEMENTS YES 

I used the adjuncts with most of the reading 

assignments. 

I used the adjuncts only to study for tests. 

I read the reading assignment before looking 

at the adjunct questions. 

I preferred the multiple choice adjunct 

questions to the constructed response. 

When my section didn't have the adjuncts 
I looked at the adjuncts from another 

section. 

I thought the adjuncts helped me to study the 

reading assignment. 

I would, like to have adjunct questions for 

other classes. 

I would have learned the material better 
studying my own way without the adjunct 

materials. - I 

The adjunct questions did not help or hinder 

me in studying. 

The adjunct questions were a waste of my time. 
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APPENDIX C: DATA ANALYSIS 



t 
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C-i INTRODUCTION 

A multiple regression analysis was used to analyze the results 
of the study. Details of the analysis used are described In the 
following sections. 

C-2 PREDICTION VECTORS / 

The data from the four experiments were coded into four matrlcles 
for use In the analysis. To facilitate coding, unique numbers were 
assigned to each treatment-section combination. The numbering system 
Is shown In Figures CI and C2. The matrlcles contained the following 
vectors: 

Y = the crlterloi. score 

= 1 If the criterion score Is from treatment-section 

combination 1, 0 if otherwise 
X2 = 1 if the criterion score is from treatment-section 

combination 2, 0 if otherwise 



X = 1 if the criterion score is from treatment-section 
J 

combination j, 0 if otherwise 
P^ = 1 if the criterion score is from person 1, 0 
if otherwise 



P = 1 if the criterion score is from person n, 
n 

0 if otherwise 
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Treatments 





1 


2 


3 


1 


1 


i 

2 


3 


o 
Z 




5 


6 


3 


7 


8 


9 1 

i 


• 




• 




! 

• 






j-3 


k 




j-1 


j 









j « The total number of treatment-section 
combinations 

Figure CI . Numbers Assigned to Treatment-Section Combinations 
Experiments 1 and 2. 

Treatment 





1 






2 


1 


1 


1 

t 2 


2 


3 


4 


3 


5 






6 


• 


• 


• 


k 


j-1 


j 



j • The total number of treatment-section 
combinations 

Figure C2 . Numbers Assigned to Treatment-Section 
Combinations for Experiments 3 and 4, 
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c-3 FULL MODEL • 

The full model used in the analysis was: 

Y . a^U + a^X^ + a^X^ + • • a.X. + b^P^ + • • b^P + 

(Model 1) 

C-4 QUESTIONS TO BE ANSWERED 

The questions of the study were restated in a more explicit 
form to provide a guide for the statistical analysis • The questions 
and regression models developed to answer the questions are given 
in this section. The same basic questions were asked for all four 
experiments. The regression equations were of the same basic form 
for the four experiments but varied in the number of prediction vectors 
due to the different numbers of sections and treatments used in the 
experiments. 

Question 1: Is there an interaction between treatments and sections 
for the relevant items of the criterion test (i.e., 
is the effect of each treatment the same for all 
sections)? 

Question 2: Is there an interaction between treatments and sections 
for the incidental items of the criterion test Ci-e*, 
is the effect of each treatment the same for all sections) 
If interaction is present, the influence of the treatments will 
not be the same for different sections. If there is no interaction the 
treatments will have the same relative effect for all sections. The 
no interaction hypothesis can be stated: 
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^ 

Since the value of each mean as estimated from the regression inndel 
is a function of the associated weights for each variable, the 
hypothesis can be rewritten: 

^1 - ^ = ^2 - ^5 " ^3 - ^5 

^ - ^7 = ^5 - ^8 = ^ - ^ 

This Implies that: 

- = ^2 ~ ^5 " "^2 

^4 ~ ^7 " "^l ^5 " ^8 " "^2 

) 

V5"^j-2 = ''l V4"^j-l""'2 
and 

a^ = + a^ ^2 = '^2 ^5 

% = "^1 + ^7 - '^5 ^ ^2 ^8 

• * 

The full model can be restricted to reflect the null hypothesis by 
substituting the above values for a- , a , ...a . into the model. 
This yields the restricted model: 

+. • .+ (C;L+a^.2)Xj_5+(c2+aj_3^)Xj^^+agXg+. . .-hh^?^-^. . . 

+b P 
n n 2 

Which simplifies to: 
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Where: 

= 1 if Treatment 1, 0 otherwise 
= 1 if Treatment 3, 0 otherwise 
S = 1 if Section 1, 0 otherwise 



^ = 1 if section k, 0 otherwise 
However, examination of the section and subject vectors reveals that the 
section vectors do not add any new information since knowledge of the 
subject includes knowledge of his section. Therefore, the section vectors 
are redundant and can be eliminated from the model. The model may be 
rewritten as: 



Where: 

Tj^= 1 if Treatment 1, 0 if otherwise 

T^= 1 if Treatment 3, 0 if otherwise 
The criterion scores were predicted using Models 1 and 2. The significance 
of the interaction was then tested by computiYig F from the RSOs for the 
two models. 

Question 3: Are the mean scores of the subjects on the 



Y 




(Model 2) 



relevant items of the criterion tests significantly 



different when the subjects have used multiple 



choice adjunct or constructed response adjunct 



materials than when they have not used adjunct , 



materials? 
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Question 4: Are the mean scores of subjects on Incidental 
Items of the criterion tests significantly 
different when the subjects have used multiple 
choice or constructed response adjunct materials 
than when they have not used adjunct materials? 
Restated) the questions are: are the mean scores under treatments 
1, 2, and 3 significantly different for relevant items? For incidental 
items? The null hypothesis is: 

or in multiple regression terms 
^4= S = ^6 



or 



a, ^= a. = a, 
j-2 3-1 2 



a^-a^= 0 



or 



a. ,-a.= 0 
j-1 3 



a2=a3+c^ 
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Where: c^= 0 

Substituting Into the full model yielded the restricted model: 
Y = a^U + a2S^ + 3382+. . . a._3^S^+. . .+d^P^. . .+d^P^ + 

^ Since the section variables are redundant, they may be omitted 

from the model. The model may be rev/ritten as: 

Y = a^U + a^P^+...+a^P^ + e3 

(Model 3) 

The criterion score was predicted using Models 2 and 3. The sig- 
nificance of the difference of the mean scores was then tested by computing 
F from resulting RSQs. A significant F value obtained from this analysis 
.indicates that at least two of the treatment means are significantly diff- 
erent. 

Since there were more than two treatments used in Experiments 1 and 

2, it was not possible to determine which pair or pairs of treatment 

means were significantly different when a significant F value was obtained. 

The Scheffe technique for multiple comparisons (Ferguson, 1966) was used 

to test the significance of the difference between pairs of means. This 

was accomplished by computing F for each pair of means and comparing the 

t 

F value obtained with the adjusted F value (F ) required for significance 
by the Scheffe'' technique. Regression procedures were used to compute 
the F values for each pair of means. This was accomplished by applying 
the following restrictions to Model 2: 
Null hypotheses: X^^- Xj,^ 



= 0 



= 0 or 



= 0 or 
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Substituting the restrictions into Model 2 yields the following 
-models : 

(Model 4) 
(Model 5) 
(Model 6) 

Each model was used to predict the criterion score. The resulting RSOs 
were then used to compute F for each pair of means • 

The above analysis also provided a basis for answering two questions 
related to the second area of consideration in the study - the relative 
effect of multiple choice and constructed response questions. The questions 
were : 

Question 5: Are the mean scores of subjects on the relevant 

items of the criterion test significantly different 
when the subjects have used multiple choice adjunct 
materials than when they have used constructed 
response adjunct materials? 

Question 6: Are the mean scores of subjects on the incidental 

items of the criterion test significantly different 
when the subjects have used multiple choice adjunct 
materials than when they have used constructed 
response adjunct questions? 

C-5 ADDITIONAL ANALYSIS 

A further analysis was made to provide additional information for 
evaluating the effects of adjunct materials. In this analysis, the 
proportion of variance accounted for by two additional variables, section 



and Instructor, was computed. Information on these variables was 
Indirectly Included In the regression analysis as part of the subject 
variable. However, It was not possible to determine the Influence of 
the section and Instructor variables from this analysis. 

The proportion of variance accounted for by each variable was computed 
by developing two additional regression models. They were: 

Y = a^U+ b-I>. . -^-b, L + d P^+. . .-H P + e- 
0 11 kkll nn5 

(Model 7) 

and Y = a^U+ C-S-+. . .+c^S^+. . .+d^P^+. • .d +P + e. 

0 11 jj 11 nn6 

(Model 8) 

Where: 

Ij^* 1 If the criterion score Is from Instructor 1, 0 If otherwise 



1 If the criterion score Is from instructor k, 0 if otherwise 
Sj^= 1 if the criterion score is from section 1, 0 if otherwise 



Sj= 1 if the criterion score is from section j, 0 if otherwise. 

In addition, the proportion of variance accounted for by the other 
variables in the study were summarized. They were: 
Treatment (RSQ Model 2 minus RSQ Model 3) 

Treatment-Section Interaction (RSQ Model 1 minus RSQ Model 2) 
Subject (RSQ for Model 3) 



It should be noted that the proportion of variance accounted 
for by the subject variable can be divided into three parts - Variance 
due to instructors, variance due to sections^_ and variance due to 
unidentified subject factors (such as aptitude, motivation, etc,)- It 
should also be noted that knowledge of section implies knowledge of 
instructor. Therefore, the variance accounted for by instructor is 
included in the variance accounted for by section. Similarly since 
knowledge of subject implies knowledge of instructor and section, the 
variance accounted for by instructor and section is included in the 
total subiect variance. 
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